{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How AutoDL Works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "- [Introduction](#Introduction)\n",
    "- [Implementation in arcgis.learn](#Implementation)\n",
    "    - [Imports](#imports)\n",
    "    - [Prepare data](#prepare)\n",
    "- [Train networks using AutoDL](#AutoDL)\n",
    "    - [AutoDL Training modes](#ModesAutoDL)\n",
    "- [Supported methods in AutoDL](#MethodsAutoDL)\n",
    "    - [Supported Classification Models](#scm)\n",
    "    - [Supported Detection Models](#sdm)\n",
    "    - [fit](#fit)\n",
    "    - [Score](#score)\n",
    "    - [Show Results](#results)\n",
    "    - [MIOU](#miou)\n",
    "    - [Average Precision Score](#aps)\n",
    "    \n",
    "- [Fine tune AutoDL models using ImageryModel](#ImageryModel)\n",
    "    - [Load the model](#imLoad)\n",
    "    - [Learning rate](#imLR)\n",
    "    - [Train the model](#imFit)\n",
    "    - [Save the model](#imSave)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "<a id='Introduction'></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>This guide explains the steps for the training and evaluation of multiple network architectures supported by the <code>arcgis.learn</code> API. &nbsp;The <code>arcgis.learn</code> API currently supports more than 30 deep learning networks for object detection. The API provides 4 deep learning networks, along with the <code>MMDetection</code> class, which supports more than twenty object detection networks. Similarly, for pixel classification, the API supports 11 deep learning networks, along with the <code>MMSegmentation</code> class, which supports more than twenty pixel classification networks. </p><p>To train a deep learning network using the <code>arcgis.learn</code> API, you must follow the complete pipeline, which involves data preprocessing, network selection, hyper parameter tuning, and network selection/evaluation based on the performance of the network. </p><p>As it can be difficult to iteratively run and compare all of the different networks, the <code>AutoDL</code> class automatically trains all of the supported networks with the given data within a provided time limit and provides a performance tally for all of the networks. The <code>AutoDL</code> class will also save all of the networks during the training process, allowing them to be used later for fine tuning to enhance the network performance. </p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AutoDL supported Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Object Detection**\n",
    "\n",
    "- SingleShotDetector\n",
    "- RetinaNet\n",
    "- FasterRCNN\n",
    "- YOLOv3\n",
    "- ATSS\n",
    "- CARAFE\n",
    "- CascadeRCNN\n",
    "- CascadeRPN\n",
    "- DCN\n",
    "- Detectors\n",
    "- DoubleHeads\n",
    "- DynamicRCNN\n",
    "- EmpiricalAttention\n",
    "- FCOS\n",
    "- FoveaBox\n",
    "- FSAF\n",
    "- GHM\n",
    "- LibraRCNN\n",
    "- PaFPN\n",
    "- PISA\n",
    "- RegNet\n",
    "- RepPoints\n",
    "- Res2Net\n",
    "- SABL\n",
    "- VFNet\n",
    "\n",
    "**Pixel Classification**\n",
    "\n",
    "- DeepLab\n",
    "- UnetClassifier\n",
    "- PSPNetClassifier\n",
    "- ANN\n",
    "- APCNet\n",
    "- CCNet\n",
    "- CGNet\n",
    "- HRNet\n",
    "- DeepLabV3Plus\n",
    "- DMNet\n",
    "- DNLNet\n",
    "- EMANet\n",
    "- FastSCNN\n",
    "- FCN\n",
    "- GCNet\n",
    "- MobileNetV2\n",
    "- NonLocalNet\n",
    "- OCRNet\n",
    "- PSANet\n",
    "- SemFPN\n",
    "- UperNet\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation in `arcgis.learn`\n",
    "<a id='Implementation'></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how `AutoDL` class works with `arcgis.learn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports\n",
    "<a id='imports'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from arcgis.learn import AutoDL, prepare_data, ImageryModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare data\n",
    "<a id='prepare'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the data for `AutoDL` class using `prepare_data()` in `arcgis.learn`, the recommended value for the batch_size parameter is `None` as `AutoDL` class supports automatic evaluation of the batch_size based on the GPU capacity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = prepare_data(\"path_to_data_folder\", batch_size=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train networks using AutoDL\n",
    "<a id='AutoDL'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`AutoDL` class accepts the following paramters:\n",
    "\n",
    "- `data` (Required Parameter): The data object returned from the `prepare_data` function in the previous step.\n",
    "    \n",
    "- `total_time_limit` (Optional parameter): The total time limit in hours for the `AutoDL` to train and evaluate the networks. This parameter becomes important when time is the main constraint to the user. The `AutoDL` class calculates the number of chips that can be processed in the given time from the prepared databunch.\n",
    "    \n",
    "    \n",
    "- `mode` (Optional Parameter): Can be \"basic\" or \"advanced\".\n",
    "\n",
    "    - _basic_ : To to be used when the user wants to train all selected networks.\n",
    "    - _advanced_ : To be used when the user also wants to tune hyper parameters of the two best performing models from the basic mode.\n",
    "    \n",
    "\n",
    "- `network` (Optional Parameter):\n",
    "    The list of models that will be used in the training process. If the user does not provide the parameter value, the `AutoDL` class selects all of the supported networks, however the user can select one or more networks by passing the network names as string in a list.\n",
    "\n",
    "    - _Supported Object Detection models_\n",
    "        - SingleShotDetector, RetinaNet, FasterRCNN, YOLOv3, ATSS, CARAFE, CascadeRCNN, CascadeRPN, DCN, Detectors, DoubleHeads, DynamicRCNN, EmpiricalAttention, FCOS, FoveaBox, FSAF, GHM, LibraRCNN, PaFPN, PISA, RegNet, RepPoints, Res2Net, SABL, VFNet\n",
    "        \n",
    "    - _Supported Object Detection models_\n",
    "        - DeepLab, UnetClassifier, PSPNetClassifier, ANN, APCNet, CCNet, CGNet, HRNet, DeepLabV3Plus, DMNet, DNLNet, EMANet, FastSCNN, FCN, GCNet, MobileNetV2, NonLocalNet, OCRNet, PSANet, SemFPN, UperNet\n",
    "        \n",
    "\n",
    "- `verbose` (Optional Parameter): Optional Boolean.\n",
    "    To be used to display logs while training the networks. This parameter displays the progress with time and becomes important in case of any failure, user can use the logs to check which, when and why network training failed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AutoDL Training modes\n",
    "<a id='ModesAutoDL'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Basic**\n",
    "    - In this mode we iterate through all of the supported networks once with the default backbone, train it with the passed data, and calculate the network performance. At the end of each iteration, the function will save the model to the disk. Based on the alloted time, the program will automatically calculate the maximum number of epochs to train each network. However,the training will stop if the model stops improving for 5 epochs. A minimum difference of 0.001 in the validation loss is required for it to be considered as an improvement.\n",
    "    \n",
    "    \n",
    "- **Advanced**\n",
    "    - To be used when the user wants to tune the hyper-parameters of two best performing networks from the basic mode. This mode will divide the total time into two halves. In the first half, it works like basic mode, where it will iterate through all of the supported networks once. In the second half, it checks for the two best performing networks. The program then trains the selected networks with different supported backbones. At the end of each iteration, the function will save the model to the disk. Based on the alloted time, the program will automatically calculate the maximum number of epochs to train each network. However,the training will stop if the model stops improving for 5 epochs. A minimum difference of 0.001 in the validation loss is required for it to be considered as an improvement. \n",
    "    - In this mode we use `optuna` to tune the hyper-paramaeters of the network. \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = AutoDL(data, total_time_limit=5,verbose=True, mode=\"advanced\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>When the <code>AutoDL</code> class is initialized, it calculates the number of images that can be processed in the given time and the time required to process the all of the data. The output of the cell above can then be used to analyze and update the <code>total_time_limit</code> and <code>networks</code> parameters while initializing the class. </p><p><strong>Here is an example of the output</strong> </p><ul><li>Given time to process the dataset is: 5.0 hours</li><li>Number of images that can be processed in the given time: 290</li><li>Time required to process the entire dataset of 3000 images is 52 hours</li></ul><p>This explains how many images can be processed to train all of the selected networks in the selected mode within the given time, as well as it provides an estimate of the time that <code>AutoDL</code> will take to train all of the selected networks with the entire dataset.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supported methods in AutoDL\n",
    "<a id='MethodsAutoDL'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supported Classification Models\n",
    "<a id='scm'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.supported_classification_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this function will be a list of pixel classification models supported by the `AutoDL` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supported Detection Models\n",
    "<a id='sdm'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.supported_detection_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this function will be a list of object detection models supported by the `AutoDL` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit\n",
    "<a id='fit'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fit` method will be used to train all of the selected networks automatically within the provided time limit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score\n",
    "<a id='score'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method will return an evaluation report as a dataframe that will include several fields, including the model's accuracy with train/validation loss, dice(for pixel classification), the learning rate used to train the model, train time, and backbone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.score()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Report \n",
    "<a id='report'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.report()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method will return an advanced html report of the networks trained by `AutoDL`. In the `basic` mode it shows the leaderboard of all the networks based on their performance during the training phase, and some important parameter details and charts for the best evaluated model. Additionally, in the `advanced` mode it shows details of all the `optuna` based trails with the hyper-tuned parameter details and the feature importance chart for each of the model evaluated during the `advanced` mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Results \n",
    "<a id='results'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method will display the results for the best performing model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MIOU \n",
    "<a id='miou'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MIOU is mean of intersection over union, this method calculates mean IOU on the validation set for each class. This function is only supported by pixel classification models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.mIOU()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average Precision Score \n",
    "<a id='aps'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method computes the average of the precision on the validation set for each class. This function is only supported by object detection models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl.average_precision_score()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine tune AutoDL models using ImageryModel\n",
    "<a id='ImageryModel'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the best performing network is identified, it can be further fine tuned using the `ImageryModel` class. This class supports methods that can be used to load, fine-tune, and save the model for further use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im = ImageryModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the model\n",
    "<a id='imLoad'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The load method is used to load a saved model from the disk using the `AutoDL` class. It accepts the following parameters:\n",
    "    - path: Path to the ESRI Model Definition (emd or dlpk) file\n",
    "    - data: Returned data object from `prepare_data` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im.load(\"path_to_emd_file\", data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning rate\n",
    "<a id='imLR'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `lr_find` method runs the Learning Rate Finder, which helps in choosing the optimum learning rate for training the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model\n",
    "<a id='imfit'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loaded model can be trained further using the fit method. This method trains the model for the specified number of epochs while using the specified learning rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im.fit(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the model \n",
    "<a id='imSave'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method saves the model weights and creates an Esri Model Definition and a Deep Learning Package zip for deployment to Image Server or ArcGIS Pro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im.save(\"path_to_save_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This guide has explained how the `AutoDL` class can be used to automate multiple deep learning models supported by the `arcgis.learn` API. For every step in the workflow, we defined a function and discussed its usage. This guide can be a starting point for developers to train and evaluate multiple `arcgis.learn` supported models' performances.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about the API, refer to the <a href=\"https://developers.arcgis.com/python/api-reference/arcgis.learn.toc.html#autodl\">API reference</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
